{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import scipy.cluster.hierarchy as shc\n",
    "\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import silhouette_score\n",
    "\n",
    "from sklearn import datasets\n",
    "\n",
    "%matplotlib inline\n",
    "pd.set_option(\"display.max_columns\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 22 - Determining the number of clusters\n",
    "\n",
    "We will look at two methods for determining the number of clusters.  \n",
    "\n",
    "## Inertia and the elbow method\n",
    "The first method assmes you have centers for the clusters, as in k-means clustering. It computes the sum of the squared distances of samples to their closest cluster center.\n",
    "\n",
    "We'll load the iris dataset, as in Lab 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "iris_dict = datasets.load_iris()\n",
    "\n",
    "iris = pd.DataFrame(iris_dict.data, columns = iris_dict.feature_names)\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the data columns to be between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use k-means with k = 3 to compute the clusters.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the sum of the squared distance of the samples to their closest cluster center as follows (`kmeans` should be the variable holding information about the k-means clustering algorithm)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.inertia_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the best k value, we make a loop to compute the inertia for each k, storing the result in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inertia_list = []\n",
    "for k in range(1,11):\n",
    "    kmeans = KMeans(n_clusters=k, random_state=0)\n",
    "    kmeans_clusters = kmeans.fit_predict(iris_scaled)\n",
    "    inertia_list.append(kmeans.inertia_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the values in inertia_list.  You can use `range(1,11)` as the x values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The elbow method tells us to look for where the curve straightens into a line.  That point is the suggested number of clusters.\n",
    "\n",
    "We'll try this approach to determine the cluster number for the labor market data.  Let's load in and clean the labor market data from the previous labs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labor = pd.read_csv(\"../data/Feb2019_labor_market_majors.csv\", skiprows = 13, \\\n",
    "                    skipfooter = 3, index_col = \"Major\")\n",
    "labor[\"Median Wage Early Career\"] = labor[\"Median Wage Early Career\"].str.replace(\",\",\"\").astype(float)\n",
    "labor[\"Median Wage Mid-Career\"] = labor[\"Median Wage Mid-Career\"].str.replace(\",\",\"\").astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labor.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new dataframe with the scaled data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run k-means clustering on the scaled data with k = 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the inertia for 4 clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the inertia if there is only 1 cluster?  What is the inertia if every data point is its own cluster?\n",
    "\n",
    "Compute the inertia for all values of k between 1 and 10 using a loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, plot the inertias as a line graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where do you think the elbow is for this graph?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Silhouette Score\n",
    "\n",
    "Instead of computing the inertia, which requires a cluster center, we can compute the silhouette score.  \n",
    "\n",
    "First the Silhouette Coefficient is calculated for each data point.  If a is the mean distance from that point to all other points in its cluster and if b is the mean distance to all other points in the nearest cluster that the point is not part of, then the Silhouette Coefficient for a data point is \n",
    "$$\\frac{b - a}{\\max\\{a,b\\}}$$\n",
    "\n",
    "The Silhouette Score is the mean silhouette coefficient for all data points.\n",
    "\n",
    "Again compute the k-means clusters for the iris data set with k =3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the silhouette score as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "silhouette_score(iris_scaled,kmeans_clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find the value of k giving the lowest (best) silhouette score by using a loop to try different values of k, similarly to the elbow method.  Try doing this below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you get a similar answer as with the elbow method?\n",
    "\n",
    "Now try using the silhouette score to find the best value of k for the labor data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the k giving the best silhouette score compare to the elbow method?\n",
    "\n",
    "## Starbucks drinks dataset\n",
    "\n",
    "Try both the elbow method and the silhouette score to compute the optimum number of clusters for Starbucks drinks, based on their nutritional information.  The original dataset is from Kaggle [here]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}